VIREO@TRECVID 2011: Instance Search, Semantic Indexing, Multimedia Event Detection and Known-Item Search

The vireo group participated in four tasks: instance search, semantic indexing, multimedia event detection and known-item search. In this paper,we will present our approaches and discuss the evaluation results. Instance Search (INS): We experimented four runs to contrast the following for instance search: full matching (vireo b) versus partial matching (vireo m); use of weak geometric information (vireo b) versus stronger spatial configuration (vireo s); use of face matching (vireo f). F X NO vireo b 2: Full keyframe-level matching by Bag-of-Words (BoW) retrieval with weak geometric consistency checking (WGC [19]) as post-processing. F X NO vireo s 3: Full matching by BoW retrieval and modeling of spatial configuration using Enhanced WGC (E-WGC [21]) and Geometric-preserving Visual Phrases (GVP [20]). F X NO vireo f 1: Full matching by linear fusion of F X NO vireo b 2 with face matching. F X NO vireo m 4: Partial matching by weighting the importance of instance and background context. Semantic Indexing (SIN): For concept detection, one common challenge is the scarcity of training samples. Because there is a significantly increased number of concepts being considered this year, the number of collected training samples per concept is fairly limited. To alleviate this problem, we adopt the Web image sampling algorithm named Semantic Field [10] to enrich the training set provided by TRECVID 2011. Our main focus for the SIN task is on the study of following two issues: 1) the effectiveness of models learnt from Web images on TRECVID 2011 dataset, and 2) the concept learning performance of combining training sets from TRECVID and a Web image collection.. The concept detection system is similar to our TRECVID 2009 system, where both local and global features are employed to train SVM models for each concept. We submitted four runs as summarized below: F A vireo.baseline video: Concept detectors learnt on the training set provided by TRECVID 2011 only. F B vireo.SF web image: Concept detectors learnt on the training set sampled from Web images using Semantic Field (SF) method. F D vireo.A-SVM: Using training set provided by TRECVID 2011 to update SF models based on adaptive SVM (A-SVM) [8] algorithm. F D vireo.TradBoost: Aggregation of the training sets from Web images and TRECIVD 2011 in a TradaBoost [22] learning framework. Multimedia Event Detection (MED): Framework proposed by Jiang et al. [3] is adopted as our baseline for further improvement with additional features. First of all, visual and audio features are extracted from videos. Features extracted include SIFT, ColorSIFT, MFCC and STIP. Bag-of-Word (BoW) is used to represent the features extracted and SVM is trained to classify the events. Weighted fusion is modeled to fuse the results from the classifiers of different modalities to improve the performance. Our submissions are: AutoEAG p-RUN1: STIP + MFCC + SIFT AutoEAG c-RUN2: STIP + MFCC + SIFT + ColorSIFT AutoEAG c-RUN3: STIP + MFCC Known-Item Search (KIS): Our objective for the KIS task is to observe the effectiveness of different modalities (metadata, automatic speech recognition (ASR) and concepts). We adopt the same technique we developed last year to gauge its performance on this year’s dataset. Consistent with previous year’s results, the evaluation once again shows that concept-based search is useless towards known-item search whereas textual-based modalities continue to deliver reliable performance especially the metadata. Different from previous year result, supplementing the metadata with the ASR feature is not longer able to boost the performance. We submitted four runs for the fully automatic settings as follows: F A YES vireo run1 metadata asr 1: metadata + ASR. F A YES vireo run2 metadata 2: metadata only. F A YES vireo run3 asr 3: ASR only. F A YES vireo run4 concept 4: concept only.